Methods of Data Analysis Working with probability distributions

One of the key problems in non-parametric data analysis is to create a good model of a generating probability distribution, assuming we are given as data a finite sample from that distribution. Obviously this problem is ill-posed for continuous distributions: with finite data, there is no way to distinguish between (or exclude) distributions that are not restricted to be smooth. The question then becomes of how to formulate the problem where a finite data set can be used to choose the “best” distribution among the distributions that are “sufficiently smooth”, where, intuitively, more data should allow us to consider less smooth distributions. A popular way of constructing such continuous distribution estimates is by kernel density estimation (KDE). Another method is to describe the data with a parametric family of distributions that is sufficiently rich so that, in the limit of large data, it can describe an arbitrary distribution. A well-known example from this class are Mixtures of Gaussians. Another versatile framework for modeling both continuous and discrete distributions of potentially high dimensionality given a finite sample is by means of maximum entropy (ME) approach. Here, we are looking for the most random distribution (= maximum entropy) that exactly reproduces a chosen set of statistics which can reliably be estimated from data. The assumption of maximum entropy is a formal version of Occam’s razor: one chooses distributions of a particular form that contain a minimal amount of structure that is nevertheless sufficient to explain selected aspects of observations (constraints).